Cloud access security broker systems and methods with an in-memory data store

ABSTRACT

A method performed by a Cloud Access Security Broker (CASB) service includes scanning data stored in one of a cloud provider and a Software-as-a-Service (SaaS) application, wherein the data is for a user associated with a company of a plurality of companies; detecting an incident in a file or email in the data during the scanning; maintaining details of the incident in an in-memory data store, including a current snapshot of the file or email; and providing a notification to the tenant of the incident. The method can further include, subsequent to the incident and while the file or email is being updated, updating the details of the incident in the in-memory data store.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to networking and computing.More particularly, the present disclosure relates to Cloud AccessSecurity Broker (CASB) systems and methods with an in-memory data store.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to networking and computing.More particularly, the present disclosure relates to Cloud AccessSecurity Broker (CASB) systems and methods with an in-memory data store.

BACKGROUND OF THE DISCLOSURE

Traditionally, before the cloud, corporate or enterprise resources werefully under the control of Information Technology (IT) administration(“admins”). That is, sensitive enterprise data was located within anetwork under IT admin control with perimeter defenses. Here, IT adminshave full control of access privileges, activity, etc. As is well-known,enterprises are moving their IT infrastructure to the cloud for avariety of cloud services (Software-as-a-Service (SaaS)) for email(e.g., Office 365, Gmail, etc.), file storage (OneDrive, Dropbox, Box,Google Drive, SharePoint, etc.), document preparation and contentcollaboration (e.g., Office 365, Google Docs, etc.), CustomerRelationship Management (CRM) (e.g., Salesforce, etc.), and the like.Here, enterprise IT admins no longer have the same level of control ofenterprise resources, i.e., content is stored in the cloud, and ITsimply does not have the same level of control as before.

A Cloud Access Security Broker (CASB) is an on-premises system orcloud-based service between cloud service users and cloud applications.The CASB is configured to monitor activity and enforce securitypolicies, such as monitoring user activity, warning administrators aboutpotentially hazardous actions, Data Loss Prevention (DLP), enforcingsecurity policy compliance, automatically preventing malware, etc. Forexample, a CASB system, either on-premises or as a cloud-based service,can scan through a large number of files in a cloud or SaaS application,e.g., Office 365, Dropbox, Box, Google Drive, Salesforce, etc. Thisplaces tremendous loads on traditional CASB systems, resulting inlatency, inability to properly scan all files, poor user experience,etc. In effect, an objective of a CASB system or scanner is to provideIT admin with control as if the enterprise resources were fully underthe IT admin's control as before the cloud.

One particular problem for enterprises and their SaaS services is theso-called “shadow ID” problem. Here, enterprise users use their userID/password for SaaS services when registering at third party sites.That is, users typically use their email (as user ID) and reusepasswords. As such, login credentials for the SaaS services may becompromised, leaving a hole in enterprise resource security. Forenterprises, with users distributed geographically and with multipleSaaS applications, it is critical to provide CASB protection as soon aspossible, namely scanning data and preventing data loss at nearreal-time is critical. In this context, there is a need to identifychanges, active users, etc. and a need to load balance activity, all inan effort for near real-time scanning of SaaS applications.

A CASB system can operate through an Application Programming Interface(API) provided by a cloud or SaaS provider. The CASB system can includedetection of DLP violations and/or malware in files, email, etc., whichcan be referred to as an incident. There is a need to captureinformation related to the incident for logging and reporting purposesin a scalable and efficient manner.

BRIEF SUMMARY OF THE DISCLOSURE

The present disclosure relates to Cloud Access Security Broker (CASB)systems and methods with an in-memory data store. This related to a CASBsystem performing scans to detect DLP violations and/or malware, withthe resultant scan result referred to as an incident. The presentdisclosure provides logging and reporting for the incidents. The loggingand reporting requirements for the incident reports are different fromregular weblog and firewall logs where the previous transactions areimmutable. For CASB incidents, a file in the cloud provider can bemodified and rescanned again and again multiple times. IT from a companywould like to see the current snapshot of the incidents and analyze theincident results. The present disclosure provides a highly scalable andefficient incident reporting approach for the latest snapshot for thescan (for each file and email).

Systems and methods include receiving a record associated with anincident that was detected by the CASB system in a Software-as-a-Service(SaaS) application; determining a hash based on a plurality of levelsfor the record; determining if the record exists in a data store basedon the hash, and if the record exists, deleting an old record; andinserting the record in the data store based on the hash, wherein thedata store is maintained in-memory and includes records at leaf nodes ina multi-level hash based on the plurality of levels. The incident can beone of a Data Loss Prevention (DLP) violation and malware, each beingdetected by a scan by the CASB system of the SaaS application. The CASBsystem can be a multi-tenant system, and wherein the plurality of levelsinclude a company identifier, an application identifier for the SaaSapplication, and a tenant name for a user, each being individuallyhashed. The systems and methods can further include periodically storingrecords in the data store in a file for recovery. The systems andmethods can further include receiving a query related to a count basedon one or more of the plurality of levels; and responding to the querybased on a count of the one or more of the plurality of levels. Thesystems and methods can further include broadcasting a delete request toother nodes in a cluster so that any node having a key based on the hashof each the plurality of levels deletes an old record. The systems andmethods can further include receiving a delete request for a recordwhere a query is running; and marking the record for deletion such thatthe record is deleted after the query.

Also, the present disclosure relates to Cloud Access Security Broker(CASB) systems and methods for active user identification and loadbalancing. In particular, the present disclosure describes an efficientCASB system that can perform distributed file crawling for a company(organization) to scan files for associated policies, take actions basedon the associated policies, provide reports/control, and can integratewith cloud-based security systems. To achieve efficiency, the presentdisclosure includes detection of changes in data in a SaaS as soon as auser is active and modifies the data, and a scan of the data at alocation closest to the source of the data, as well as scans incompliance with local law and regulations. The objective is to provide acompany's IT administration control of files and other content stored incloud applications. In an embodiment, the present disclosure utilizesWebhook integration in the CASB system to identify changes. In anotherembodiment, the present disclosure utilizes user geolocation for routingscans to a closest CASB scanner. The present disclosure is agnostic withrespect to a cloud application, operating with various different cloudapplications (SaaS) based on Application Programming Interfaces (APIs).Also, the present disclosure includes a so-called “assembly line”approach where various workers operate in parallel to efficientlyprocess content through various queues, including various hand-offs. TheCASB system described herein does not store customer data permanently,nor does it store confidential credentials, and the CASB system supportsenormous scale (e.g., billions of files or more) along with a configuredthrottle rate by the cloud applications.

Systems and methods include causing a scan by Cloud Access SecurityBroker (CASB) system of a plurality of users associated with a company(organization) in a Software-as-a-Service (SaaS) application where thescan includes any of identifying malware in content in the SaaSapplication and identifying confidential data in the content in the SaaSapplication; during the scan which is covering historical data in theSaaS application, receiving notifications of the content being activelymodified by any of the plurality of users; and including the contentbeing actively modified in the scan with the historical data. Thesystems and methods can further include maintaining geolocation of theany of the plurality of users; and causing the content being activelymodified in the scan to be processed by the CASB system based on thegeolocation.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated and described herein withreference to the various drawings, in which like reference numbers areused to denote like system components/method steps, as appropriate, andin which:

FIG. 1A is a network diagram of a cloud-based system offering securityas a service;

FIG. 1B is a network diagram of an example implementation of thecloud-based system of FIG. 1 ;

FIG. 2 is a block diagram of a server which may be used in thecloud-based system of FIG. 1 , to implement a CASB system or the like;

FIG. 3 is a block diagram of a mobile device which may be used in thecloud-based system of FIG. 1 or the like;

FIG. 4 is a network diagram of a CASB system;

FIG. 5 is a functional block diagram of filing crawling of the SaaSprovider with the CASB system;

FIG. 6 is a flowchart of a file crawling process based on a change log;

FIG. 7 is a flowchart of a file crawling process based on breadth-firsttraversal;

FIG. 8 is a flow diagram of example operations between the CASB client,the controller, the message broker, a worker, and the SaaS provider;

FIG. 9 is a flow diagram of an architecture of a CASB-webhooks system;

FIG. 10 is a flowchart of a CASB-webhooks integration process, that maybe implemented through the CASB-webhooks system of FIG. 10 , or in otherapproaches;

FIG. 11 is a flow diagram of subscription and renewal process for theregistration step and the renewal step in the CASB-webhooks integrationprocess;

FIG. 12 is a map illustrating an example system including two CASBsystems geographically distributed and two SaaS applications alsogeographically distributed;

FIG. 13 is a flowchart of a historical and live scanning process forCASB functionality;

FIG. 14 is a block diagram of a CASB in-memory data store system;

FIG. 15 is a flowchart of a record processing process implemented in thedata store of the CASB in-memory data store system of FIG. 14 ;

FIG. 16 is a diagram of an example implementation of the filehash fromthe data store of the CASB in-memory data store system of FIG. 14 ; and

FIG. 17 is a diagram of data duplication prevention in the data store.

DETAILED DESCRIPTION OF THE DISCLOSURE

Again, the present disclosure relates to Cloud Access Security Broker(CASB) systems and methods with an in-memory data store. This related toa CASB system performing scans to detect DLP violations and/or malware,with the resultant scan result referred to as an incident. The presentdisclosure provides logging and reporting for the incidents. The loggingand reporting requirement for the incident reports are different fromregular weblog and firewall logs where the previous transactions areimmutable. For CASB incidents, a file in the cloud provider can bemodified and rescanned again and again multiple times. A company wouldlike to see the current snapshot of the incidents and analyze theincident results. The present disclosure provides a highly scalable andefficient incident reporting approach for the latest snapshot for thescan (for each file and email).

Also, the present disclosure relates to Cloud Access Security Broker(CASB) systems and methods for active user identification and loadbalancing. In particular, the present disclosure describes an efficientCASB system that can perform distributed file crawling for a company(organization) to scan files for associated policies, take actions basedon the associated policies, provide reports/control, and can integratewith cloud-based security systems. To achieve efficiency, the presentdisclosure includes detection of changes in data in a SaaS as soon as auser is active and modifies the data, and a scan of the data at alocation closest to the source of the data, as well as scans incompliance with local law and regulations. The objective is to provide acompany's IT administration control of files and other content stored incloud applications. In an embodiment, the present disclosure utilizesWebhook integration in the CASB system to identify changes. In anotherembodiment, the present disclosure utilizes user geolocation for routingscans to a closest CASB scanner. The present disclosure is agnostic withrespect to a cloud application, operating with various different cloudapplications (SaaS) based on Application Programming Interfaces (APIs).Also, the present disclosure includes a so-called “assembly line”approach where various workers operate in parallel to efficientlyprocess content through various queues, including various hand-offs. TheCASB system described herein does not store customer data permanently,nor does it store confidential credentials, and the CASB system supportsenormous scale (e.g., billions of files or more) along with a configuredthrottle rate by the cloud applications.

Example Cloud-Based System Architecture

FIG. 1A is a network diagram of a cloud-based system 100 offeringsecurity as a service. Specifically, the cloud-based system 100 canoffer a Secure Internet and Web Gateway as a service to various users102, as well as other cloud services. In this manner, the cloud-basedsystem 100 is located between the users 102 and the Internet as well asany cloud services 106 (or applications) accessed by the users 102. Assuch, the cloud-based system 100 provides inline monitoring inspectingtraffic between the users 102, the Internet 104, and the cloud services106, including Secure Sockets Layer (SSL) traffic. The cloud-basedsystem 100 can offer access control, threat prevention, data protection,etc. The access control can include a cloud-based firewall, cloud-basedintrusion detection, Uniform Resource Locator (URL) filtering, bandwidthcontrol, Domain Name System (DNS) filtering, etc. The threat preventioncan include cloud-based intrusion prevention, protection againstadvanced threats (malware, spam, Cross-Site Scripting (XSS), phishing,etc.), cloud-based sandbox, antivirus, DNS security, etc. The dataprotection can include Data Loss Prevention (DLP), cloud applicationsecurity such as via Cloud Access Security Broker (CASB), file typecontrol, etc.

The cloud-based firewall can provide Deep Packet Inspection (DPI) andaccess controls across various ports and protocols as well as beingapplication and user aware. The URL filtering can block, allow, or limitwebsite access based on policy for a user, group of users, or entireorganization, including specific destinations or categories of URLs(e.g., gambling, social media, etc.). The bandwidth control can enforcebandwidth policies and prioritize critical applications such as relativeto recreational traffic. DNS filtering can control and block DNSrequests against known and malicious destinations.

The cloud-based intrusion prevention and advanced threat protection candeliver full threat protection against malicious content such as browserexploits, scripts, identified botnets and malware callbacks, etc. Thecloud-based sandbox can block zero-day exploits (just identified) byanalyzing unknown files for malicious behavior. Advantageously, thecloud-based system 100 is multi-tenant and can service a large volume ofthe users 102. As such, newly discovered threats can be promulgatedthroughout the cloud-based system 100 for all tenants practicallyinstantaneously. The antivirus protection can include antivirus,antispyware, antimalware, etc. protection for the users 102, usingsignatures sourced and constantly updated. The DNS security can identifyand route command-and-control connections to threat detection enginesfor full content inspection.

The DLP can use standard and/or custom dictionaries to continuouslymonitor the users 102, including compressed and/or SSL-encryptedtraffic. Again, being in a cloud implementation, the cloud-based system100 can scale this monitoring with near-zero latency on the users 102.The cloud application security can include CASB functionality todiscover and control user access to known and unknown cloud services106. The file type controls enable true file type control by the user,location, destination, etc. to determine which files are allowed or not.A description of DLP functionality is provided in commonly-assigned U.S.patent application Ser. No. 16/923,225, filed Jul. 8, 2020, and entitled“Data Loss Prevention via Indexed Document Management,” the contents ofwhich are incorporated by reference herein in their entirety.

For illustration purposes, the users 102 of the cloud-based system 100can include a mobile device 110, a headquarters (HQ) 112 which caninclude or connect to a data center (DC) 114, Internet of Things (IoT)devices 116, a branch office/remote location 118, etc., and eachincludes one or more user devices (an example user device 300 isillustrated in FIG. 3 ). The devices 110, 116, and the locations 112,114, 118 are shown for illustrative purposes, and those skilled in theart will recognize there are various access scenarios and other users102 for the cloud-based system 100, all of which are contemplatedherein. The users 102 can be associated with a tenant, which may includean enterprise, a corporation, an organization, etc. That is, a tenant orcompany is a group of users who share a common access with specificprivileges to the cloud-based system 100, a cloud service, etc. In anembodiment, the headquarters 112 can include an enterprise's networkwith resources in the data center 114. The mobile device 110 can be aso-called road warrior, i.e., users that are off-site, on-the-road, etc.Further, the cloud-based system 100 can be multi-tenant, with eachtenant having its own users 102 and configuration, policy, rules, etc.One advantage of the multi-tenancy and a large volume of users is thezero-day/zero-hour protection in that a new vulnerability can bedetected and then instantly remediated across the entire cloud-basedsystem 100. The same applies to policy, rule, configuration, etc.changes—they are instantly remediated across the entire cloud-basedsystem 100. As well, new features in the cloud-based system 100 can alsobe rolled up simultaneously across the user base, as opposed toselective and time-consuming upgrades on every device at the locations112, 114, 118, and the devices 110, 116.

Logically, the cloud-based system 100 can be viewed as an overlaynetwork between users (at the locations 112, 114, 118, and the devices110, 106) and the Internet 104 and the cloud services 106. Previously,the IT deployment model included enterprise resources and applicationsstored within the data center 114 (i.e., physical devices) behind afirewall (perimeter), accessible by employees, partners, contractors,etc. on-site or remote via Virtual Private Networks (VPNs), etc. Thecloud-based system 100 is replacing the conventional deployment model.The cloud-based system 100 can be used to implement these services inthe cloud without requiring the physical devices and management thereofby enterprise IT administrators. As an ever-present overlay network, thecloud-based system 100 can provide the same functions as the physicaldevices and/or appliances regardless of geography or location of theusers 102, as well as independent of platform, operating system, networkaccess technique, network access provider, etc.

There are various techniques to forward traffic between the users 102 atthe locations 112, 114, 118, and via the devices 110, 116, and thecloud-based system 100. Typically, the locations 112, 114, 118 can usetunneling where all traffic is forward through the cloud-based system100. For example, various tunneling protocols are contemplated, such asGeneric Routing Encapsulation (GRE), Layer Two Tunneling Protocol(L2TP), Internet Protocol (IP) Security (IPsec), customized tunnelingprotocols, etc. The devices 110, 116 can use a local application thatforwards traffic, a proxy such as via a Proxy Auto-Config (PAC) file,and the like. A key aspect of the cloud-based system 100 is all trafficbetween the users 102 and the Internet 104 or the cloud services 106 isvia the cloud-based system 100. As such, the cloud-based system 100 hasvisibility to enable various functions, all of which are performed offthe user device in the cloud.

The cloud-based system 100 can also include a management system 120 fortenant access to provide global policy and configuration as well asreal-time analytics. This enables IT administrators to have a unifiedview of user activity, threat intelligence, application usage, etc. Forexample, IT administrators can drill-down to a per-user level tounderstand events and correlate threats, to identify compromiseddevices, to have application visibility, and the like. The cloud-basedsystem 100 can further include connectivity to an Identity Provider(IDP) 122 for authentication of the users 102 and to a SecurityInformation and Event Management (SIEM) system 124 for event logging.The system 124 can provide alert and activity logs on a per-user 102basis.

FIG. 1B is a network diagram of an example implementation of thecloud-based system 100. In an embodiment, the cloud-based system 100includes a plurality of enforcement nodes (EN) 150, labeled asenforcement nodes 150-1, 150-2, 150-N, interconnected to one another andinterconnected to a central authority (CA) 152. The nodes 150, 152,while described as nodes, can include one or more servers, includingphysical servers, virtual machines (VM) executed on physical hardware,etc. That is, a single node 150, 152 can be a cluster of devices. Anexample of a server is illustrated in FIG. 1B. The cloud-based system100 further includes a log router 154 that connects to a storage cluster156 for supporting log maintenance from the enforcement nodes 150. Thecentral authority 152 provide centralized policy, real-time threatupdates, etc. and coordinates the distribution of this data between theenforcement nodes 150. The enforcement nodes 150 provide an onramp tothe users 102 and are configured to execute policy, based on the centralauthority 152, for each user 102. The enforcement nodes 150 can begeographically distributed, and the policy for each user 102 followsthat user 102 as he or she connects to the nearest (or other criteria)enforcement node 150.

The enforcement nodes 150 are full-featured secure internet gatewaysthat provide integrated internet security. They inspect all web trafficbi-directionally for malware and enforce security, compliance, andfirewall policies, as described herein. In an embodiment, eachenforcement node 150 has two main modules for inspecting traffic andapplying policies: a web module and a firewall module. The enforcementnodes 150 are deployed around the world and can handle hundreds ofthousands of concurrent users with millions of concurrent sessions.Because of this, regardless of where the users 102 are, they can accessthe Internet 104 from any device, and the enforcement nodes 150 protectthe traffic and apply corporate policies. The enforcement nodes 150 canimplement various inspection engines therein, and optionally, sendsandboxing to another system. The enforcement nodes 150 includesignificant fault tolerance capabilities, such as deployment inactive-active mode to ensure availability and redundancy as well ascontinuous monitoring.

In an embodiment, customer traffic is not passed to any other componentwithin the cloud-based system 100, and the enforcement nodes 150 can beconfigured never to store any data to disk. Packet data is held inmemory for inspection and then, based on policy, is either forwarded ordropped. Log data generated for every transaction is compressed,tokenized, and exported over secure TLS connections to the log routers154 that direct the logs to the storage cluster 156, hosted in theappropriate geographical region, for each organization.

The central authority 152 hosts all customer (tenant) policy andconfiguration settings. It monitors the cloud and provides a centrallocation for software and database updates and threat intelligence.Given the multi-tenant architecture, the central authority 152 isredundant and backed up in multiple different data centers. Theenforcement nodes 150 establish persistent connections to the centralauthority 152 to download all policy configurations. When a new userconnects to an enforcement node 150, a policy request is sent to thecentral authority 152 through this connection. The central authority 152then calculates the policies that apply to that user 102 and sends thepolicy to the enforcement node 150 as a highly compressed bitmap.

Once downloaded, a tenant's policy is cached until a policy change ismade in the management system 120. When this happens, all of the cachedpolicies are purged, and the enforcement nodes 150 request the newpolicy when the user 102 next makes a request. In an embodiment, theenforcement node 150 exchange “heartbeats” periodically, so allenforcement nodes 150 are informed when there is a policy change. Anyenforcement node 150 can then pull the change in policy when it sees anew request.

The cloud-based system 100 can be a private cloud, a public cloud, acombination of a private cloud and a public cloud (hybrid cloud), or thelike. Cloud computing systems and methods abstract away physicalservers, storage, networking, etc., and instead offer these as on-demandand elastic resources. The National Institute of Standards andTechnology (NIST) provides a concise and specific definition whichstates cloud computing is a model for enabling convenient, on-demandnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe rapidly provisioned and released with minimal management effort orservice provider interaction. Cloud computing differs from the classicclient-server model by providing applications from a server that areexecuted and managed by a client's web browser or the like, with noinstalled client version of an application required. Centralizationgives cloud service providers complete control over the versions of thebrowser-based and other applications provided to clients, which removesthe need for version upgrades or license management on individual clientcomputing devices. The phrase “Software as a Service” (SaaS) issometimes used to describe application programs offered through cloudcomputing. A common shorthand for a provided cloud computing service (oreven an aggregation of all existing cloud services) is “the cloud.” Thecloud-based system 100 is illustrated herein as an example embodiment ofa cloud-based system, and other implementations are also contemplated.

As described herein, the terms cloud services and cloud applications maybe used interchangeably. The cloud service 106 is any service madeavailable to users on-demand via the Internet, as opposed to beingprovided from a company's on-premises servers. A cloud application, orcloud app, is a software program where cloud-based and local componentswork together. The cloud-based system 100 can be utilized to provideexample cloud services, including Zscaler Internet Access (ZIA), ZscalerPrivate Access (ZPA), and Zscaler Digital Experience (ZDX), all fromZscaler, Inc. (the assignee and applicant of the present application).The ZIA service can provide the access control, threat prevention, anddata protection described above with reference to the cloud-based system100. ZPA can include access control, microservice segmentation, etc. TheZDX service can provide monitoring of user experience, e.g., Quality ofExperience (QoE), Quality of Service (QoS), etc., in a manner that cangain insights based on continuous, inline monitoring. For example, theZIA service can provide a user with Internet Access, and the ZPA servicecan provide a user with access to enterprise resources instead oftraditional Virtual Private Networks (VPNs), namely ZPA provides ZeroTrust Network Access (ZTNA). Those of ordinary skill in the art willrecognize various other types of cloud services 106 are alsocontemplated. Also, other types of cloud architectures are alsocontemplated, with the cloud-based system 100 presented for illustrationpurposes.

Other cloud services can include Office 365, Dropbox, Box, Google Drive,Salesforce, and the like. In the context of these services, a providerof such cloud services can be referred to as a cloud provider, a SaaSprovider, etc., and may utilize a hardware architecture similar to thecloud-based system 100. Of course, other types of cloud architecturesare also contemplated.

Example Server Architecture

FIG. 2 is a block diagram of a server 200, which may be used in thecloud-based system 100, in a CASB system, in other systems, orstandalone. For example, the enforcement nodes 150 and the centralauthority 152 may be formed as one or more of the servers 200. Theserver 200 may be a digital computer that, in terms of hardwarearchitecture, generally includes a processor 202, input/output (I/O)interfaces 204, a network interface 206, a data store 208, and memory210. It should be appreciated by those of ordinary skill in the art thatFIG. 3 depicts the server 200 in an oversimplified manner, and apractical embodiment may include additional components and suitablyconfigured processing logic to support known or conventional operatingfeatures that are not described in detail herein. The components (202,204, 206, 208, and 210) are communicatively coupled via a localinterface 212. The local interface 212 may be, for example, but notlimited to, one or more buses or other wired or wireless connections, asis known in the art. The local interface 212 may have additionalelements, which are omitted for simplicity, such as controllers, buffers(caches), drivers, repeaters, and receivers, among many others, toenable communications. Further, the local interface 212 may includeaddress, control, and/or data connections to enable appropriatecommunications among the aforementioned components.

The processor 202 is a hardware device for executing softwareinstructions. The processor 202 may be any custom made or commerciallyavailable processor, a Central Processing Unit (CPU), an auxiliaryprocessor among several processors associated with the server 200, asemiconductor-based microprocessor (in the form of a microchip orchipset), or generally any device for executing software instructions.When the server 200 is in operation, the processor 202 is configured toexecute software stored within the memory 210, to communicate data toand from the memory 210, and to generally control operations of theserver 200 pursuant to the software instructions. The I/O interfaces 204may be used to receive user input from and/or for providing systemoutput to one or more devices or components.

The network interface 206 may be used to enable the server 200 tocommunicate on a network, such as the Internet 104. The networkinterface 206 may include, for example, an Ethernet card or adapter(e.g., 10BaseT, Fast Ethernet, Gigabit Ethernet, 10 GbE) or a WirelessLocal Area Network (WLAN) card or adapter (e.g., 802.11a/b/g/n/ac). Thenetwork interface 206 may include address, control, and/or dataconnections to enable appropriate communications on the network. A datastore 208 may be used to store data. The data store 208 may include anyof volatile memory elements (e.g., random access memory (RAM, such asDRAM, SRAM, SDRAM, and the like)), nonvolatile memory elements (e.g.,ROM, hard drive, tape, CDROM, and the like), and combinations thereof.Moreover, the data store 208 may incorporate electronic, magnetic,optical, and/or other types of storage media. In one example, the datastore 208 may be located internal to the server 200, such as, forexample, an internal hard drive connected to the local interface 212 inthe server 200. Additionally, in another embodiment, the data store 208may be located external to the server 200 such as, for example, anexternal hard drive connected to the I/O interfaces 204 (e.g., SCSI orUSB connection). In a further embodiment, the data store 208 may beconnected to the server 200 through a network, such as, for example, anetwork-attached file server.

The memory 210 may include any of volatile memory elements (e.g., randomaccess memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatilememory elements (e.g., ROM, hard drive, tape, CDROM, etc.), andcombinations thereof. Moreover, the memory 210 may incorporateelectronic, magnetic, optical, and/or other types of storage media. Notethat the memory 210 may have a distributed architecture, where variouscomponents are situated remotely from one another but can be accessed bythe processor 202. The software in memory 210 may include one or moresoftware programs, each of which includes an ordered listing ofexecutable instructions for implementing logical functions. The softwarein the memory 210 includes a suitable Operating System (O/S) 214 and oneor more programs 216. The operating system 214 essentially controls theexecution of other computer programs, such as the one or more programs216, and provides scheduling, input-output control, file and datamanagement, memory management, and communication control and relatedservices. The one or more programs 216 may be configured to implementthe various processes, algorithms, methods, techniques, etc. describedherein.

It will be appreciated that some embodiments described herein mayinclude one or more generic or specialized processors (“one or moreprocessors”) such as microprocessors; Central Processing Units (CPUs);Digital Signal Processors (DSPs): customized processors such as NetworkProcessors (NPs) or Network Processing Units (NPUs), Graphics ProcessingUnits (GPUs), or the like; Field Programmable Gate Arrays (FPGAs); andthe like along with unique stored program instructions (including bothsoftware and firmware) for control thereof to implement, in conjunctionwith certain non-processor circuits, some, most, or all of the functionsof the methods and/or systems described herein. Alternatively, some orall functions may be implemented by a state machine that has no storedprogram instructions, or in one or more Application-Specific IntegratedCircuits (ASICs), in which each function or some combinations of certainof the functions are implemented as custom logic or circuitry. Ofcourse, a combination of the aforementioned approaches may be used. Forsome of the embodiments described herein, a corresponding device inhardware and optionally with software, firmware, and a combinationthereof can be referred to as “circuitry configured or adapted to,”“logic configured or adapted to,” etc. perform a set of operations,steps, methods, processes, algorithms, functions, techniques, etc. ondigital and/or analog signals as described herein for the variousembodiments.

Moreover, some embodiments may include a non-transitorycomputer-readable storage medium having computer-readable code storedthereon for programming a computer, server, appliance, device,processor, circuit, etc. each of which may include a processor toperform functions as described and claimed herein. Examples of suchcomputer-readable storage mediums include, but are not limited to, ahard disk, an optical storage device, a magnetic storage device, aRead-Only Memory (ROM), a Programmable Read-Only Memory (PROM), anErasable Programmable Read-Only Memory (EPROM), an Electrically ErasableProgrammable Read-Only Memory (EEPROM), Flash memory, and the like. Whenstored in the non-transitory computer-readable medium, software caninclude instructions executable by a processor or device (e.g., any typeof programmable circuitry or logic) that, in response to such execution,cause a processor or the device to perform a set of operations, steps,methods, processes, algorithms, functions, techniques, etc. as describedherein for the various embodiments.

Example User Device Architecture

FIG. 3 is a block diagram of a user device 300, which may be used in thecloud-based system 100 or the like. Specifically, the user device 300can form a device used by one of the users 102, and this may includecommon devices such as laptops, smartphones, tablets, netbooks, personaldigital assistants, MP3 players, cell phones, e-book readers, IoTdevices, servers, desktops, printers, televisions, streaming mediadevices, and the like. The user device 300 can be a digital device that,in terms of hardware architecture, generally includes a processor 302,I/O interfaces 304, a radio 306, a data store 308, and memory 310. Itshould be appreciated by those of ordinary skill in the art that FIG. 3depicts the user device 300 in an oversimplified manner, and a practicalembodiment may include additional components and suitably configuredprocessing logic to support known or conventional operating featuresthat are not described in detail herein. The components (302, 304, 306,308, and 302) are communicatively coupled via a local interface 312. Thelocal interface 312 can be, for example, but not limited to, one or morebuses or other wired or wireless connections, as is known in the art.The local interface 312 can have additional elements, which are omittedfor simplicity, such as controllers, buffers (caches), drivers,repeaters, and receivers, among many others, to enable communications.Further, the local interface 312 may include address, control, and/ordata connections to enable appropriate communications among theaforementioned components.

The processor 302 is a hardware device for executing softwareinstructions. The processor 302 can be any custom made or commerciallyavailable processor, a CPU, an auxiliary processor among severalprocessors associated with the user device 300, a semiconductor-basedmicroprocessor (in the form of a microchip or chipset), or generally anydevice for executing software instructions. When the user device 300 isin operation, the processor 302 is configured to execute software storedwithin the memory 310, to communicate data to and from the memory 310,and to generally control operations of the user device 300 pursuant tothe software instructions. In an embodiment, the processor 302 mayinclude a mobile-optimized processor such as optimized for powerconsumption and mobile applications. The I/O interfaces 304 can be usedto receive user input from and/or for providing system output. Userinput can be provided via, for example, a keypad, a touch screen, ascroll ball, a scroll bar, buttons, a barcode scanner, and the like.System output can be provided via a display device such as a LiquidCrystal Display (LCD), touch screen, and the like.

The radio 306 enables wireless communication to an external accessdevice or network. Any number of suitable wireless data communicationprotocols, techniques, or methodologies can be supported by the radio306, including any protocols for wireless communication. The data store308 may be used to store data. The data store 308 may include any ofvolatile memory elements (e.g., random access memory (RAM, such as DRAM,SRAM, SDRAM, and the like)), nonvolatile memory elements (e.g., ROM,hard drive, tape, CDROM, and the like), and combinations thereof.Moreover, the data store 308 may incorporate electronic, magnetic,optical, and/or other types of storage media.

The memory 310 may include any of volatile memory elements (e.g., randomaccess memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatilememory elements (e.g., ROM, hard drive, etc.), and combinations thereof.Moreover, the memory 310 may incorporate electronic, magnetic, optical,and/or other types of storage media. Note that the memory 310 may have adistributed architecture, where various components are situated remotelyfrom one another, but can be accessed by the processor 302. The softwarein memory 310 can include one or more software programs, each of whichincludes an ordered listing of executable instructions for implementinglogical functions. In the example of FIG. 3 , the software in the memory310 includes a suitable operating system 314 and programs 316. Theoperating system 314 essentially controls the execution of othercomputer programs and provides scheduling, input-output control, fileand data management, memory management, and communication control andrelated services. The programs 316 may include various applications,add-ons, etc. configured to provide end user functionality with the userdevice 300. For example, example programs 316 may include, but notlimited to, a web browser, social networking applications, streamingmedia applications, games, mapping and location applications, electronicmail applications, financial applications, and the like. In a typicalexample, the end-user typically uses one or more of the programs 316along with a network such as the cloud-based system 100.

CASB System

FIG. 4 is a network diagram of a CASB system 400. The CASB system 400can be located between the cloud-based system 100 and one or more SaaSproviders 402. As described herein, the SaaS providers 402 can bereferred to as cloud providers, cloud service providers, serviceproviders, etc. Examples of the providers 402 include, withoutlimitation, Office 365, Dropbox, Box, Google Drive, Salesforce, etc.That is the providers 402 can provide cloud services for enterprisesrelated to file sharing, document management, email, collaboration,scheduling, timekeeping, financial, etc. The key point is the enterpriseIT is moving from local applications hosted and maintained within theenterprise network to cloud-based solutions where the data is locatedoff-site, in the providers 402.

The CASB system 400 can be implemented in a cloud-based system, such asusing the architecture of the cloud-based system 100. The CASB system400 can be implemented in a private cloud, a public cloud, or a hybridcloud. Alternatively, the CASB system 400 can be one or more servers 200that can be located on-premises with an enterprise, off-premises, etc.Even further, the CASB system 400 can be collocated with the SaaSproviders 402. That is, various architecture implementations arecontemplated. Further, the CASB system 400 contemplated both operationswith the cloud-based system 100, operating as a distributed securitysystem, as well as independent operation (i.e., with the components ofthe cloud-based system 100 omitted in FIG. 4 , and with thefunctionality incorporated in the CASB system 400 itself).

The objective of the CASB system 400 is to provide enterprise IT controlover data (resources) in the SaaS providers 402. Note, as describedherein, the enterprise can be referred to as a tenant of the provider402. The CASB system 400 is configured to operate as a distributed filecrawler for files associated with a particular tenant. The CASB system400 can both provide a report based on the file crawling as well asimplement policy actions based on policy configuration.

The CASB system 400 includes one or more APIs 410, such as aRepresentational state transfer (REST) API. In an embodiment, the APIs410 connect to the cloud-based system 100, such as one of theenforcement nodes 150. Here, a user can interact with the CASB system400 via a User Interface (UI) 412 through the central authority 152.Additionally, the enforcement node 150 can connect to a log 414, such asa data store that stores statistics and transactions, for reporting. Theenforcement node 150 can also connect to a DLP engine 416 for dataleakage protection through the CASB system 400. Here, the CASB 400 canbe used to identify content, files, etc. that match sensitive data in aDLP dictionary. The user can provide policy and configuration via the UI412.

Again, the CASB system 400 can be deployed without the cloud-basedsystem 100. Here, the API 410 can connect directly to the UI 412, andthe log 414 and the DLP engine 416 can be incorporated directly in theCASB system 400, or in an external system.

The CASB system 400 includes an authentication provider 420 that isconfigured to perform authentication of the tenant with the SaaSproviders 402. The APIs 410 and the authentication provider 420 connectto a message broker 422, which is configured to interact between theAPIs 410, the authentication provider 420, and a plurality of workers430. A regulator 424 is connected to the message broker. The messagebroker 422 is a pipeline where job tickets are queued for consumption bythe workers 430.

In an embodiment, the authentication provider 420, a controller for theAPIs 410, the regulator 424, and the workers 430 are Java Springservices, and other embodiments are also contemplated. The messagebroker 422 can be a queuing service, such as using Apache Kafka,Microsoft EventHub, or other embodiments. The API controller is aliaison service that interfaces between the CASB system 400 and thecloud-based system 100.

With respect to the authentication provider 420, customer information,including tokens and credentials are not stored permanently orpersisted. Also, the CASB system 400 is not tied specifically to aparticular SaaS provider 402. That is, the CASB system 400 is configuredto operate with multiple, different SaaS providers 402. This isaccomplished through customized APIs and configuration of the workers430. Each SaaS provider 402 can have a different set of APIs andfunctionality.

The workers 430 are connected to the SaaS providers 402 and arededicated to performing particular tasks. In a sense, the plurality ofworkers 430 are organized in a pool of workers, and tasks are assignedbetween the workers 430. The CASB 400 can include a sandbox 440 that canbe connected to the DLP engine 416, and the DLP engine 416 can alsoinclude a REST API 445 connection to the SaaS providers 402. Note, thesandbox 440 can be included in the CASB system 400, or it can be anexternal system. The sandbox 440 is configured to execute files, openfiles, etc. in a safe environment to analyze whether the files aremalicious or not.

The worker pool is a collection of workers 430 that interact with theSaaS provider 402 and perform specific tasks. The pool of workers 430enables the CASB system 400 to operate efficiently in a distributednature. The workers 430 are assigned tasks from various queues, via themessage broker 422 and the regulator 424. Thus, the workers 430 canoperate akin to an assembly line, and there can be hand-offs betweenworkers 430. For example, the workers 430 can include authenticationworkers to authenticate users, tenants, etc., metadata workers toanalyze file or content metadata, file workers to scan/analyze files,action workers to perform various policy-related actions, and the like.

The workers 430 can logically be viewed as contract workers in afactory, on an assembly line, etc. The workers 430 are provided specificinstructions in a job ticket. The job ticket has information on what jobto be performed, where to get the inputs, and where to send the outputs.Every worker 430 also knows what to do when something goes wrong.

The regulator 424 is like the SCADA (Supervisory Control and DataAcquisition) in a control system architecture. The regulator 424monitors the performance of all the workers 430 and controls the overallsystem for optimum throughput.

Job Ticket Example

Again, the message broker 422 assigns jobs to the workers 430. Here isan example of a job ticket for an example job:

{ TenantID : 123456 TransactionID : 111111 JobType : GetTenantUsers RunID : 1 SaaSProvider : Google Drive ... ... ... }

Design Constraints

Again, each different SaaS provider 402 can have a different set of APIsand functionality. The CASB system 400 is configured to interface with aplurality of different SaaS providers 402. The log 414 can be configuredto store changes/events for an entire organization, including on a peruser basis.

The APIs between the CASB 400 and the SaaS providers 402 may be limited,e.g., throttled by the SaaS providers 402. As such, there is an initialbaseline crawl (i.e., a first-run) where the CASB system 400 has tocrawl and scan all files in the SaaS provider 402. This initial baselinecrawl is performed efficiently and is synchronized with the DLP engine416. After the baseline crawl, subsequent crawls are performedincrementally, namely through files that changed since the previouscrawl. For example, the first run can be referred to as run one, andeach incremental crawl is run X, which only scans and crawls files thathave changed since run X−1. In an embodiment, the period of incrementalcalls is once a day. Of course, other periods are also contemplated.

File Crawl

The SaaS providers 402 generally provide two ways to crawl through thefiles for a tenant, namely crawling based on organization-wide fileactivity or a change log and crawling based on a pseudo-breadth-firsttraversal. The file activity or a change log enables crawling based onfile changes. The pseudo-breadth-first traversal is crawling based onsnapshots.

FIG. 5 is a functional block diagram of filing crawling of the SaaSprovider 402 with the CASB system 400. Specifically, FIG. 5 illustratesfunctionality associated with file crawling in the SaaS provider 402 bythe CASB 400. The CASB 400 includes a controller 450, such as themessage broker 422 and the regulator 424. The controller 450 cancommunicate with the cloud-based system 100 and the authenticationprovider 420. The authentication provider 420 can communicate with theSaaS providers 402. The CASB 400 can also include a CASB client 460 thatincludes a worker for DLP 462 and the log 414. In the example of FIG. 5, there are edge workers 430 a that interface between the authenticationprovider 420, the SaaS provider 402, the controller 450, and the CASBclient 460. The objective of the edge workers 430 a is to perform filecrawling of the SaaS providers 402. In an embodiment, the SaaS providers402 can be file storage providers, such as, for example, Office 365(SharePoint), Box, DropBox, etc.

For illustration, an example operation is described in FIG. 5 . There isa tenant event (S1) from the controller 450 to the edge worker 430 a.The next run notification (S2) is provided from the edge worker 430 aafter all files are crawled in the run. The edge worker 430 a notes anew event (S3) with file meta-data, the edge worker 430 a fetches filedetails and provides a file for scanning (S4) which is sent to DLP 562for scanning and analysis. A policy action (S5) can be the result of theDLP 562 and provided to the edge worker 430 a. The edge worker 430 a canimplement the policy action in the SaaS provider 402 and provide theresult (S6) for the log 414. For example, a policy action can be todelete a file, quarantine a file, flag a file, etc.

Crawling Based on a Change Log

FIG. 6 is a flowchart of a file crawling process 500 based on a changelog. The file crawling process 500 contemplates implementation by theCASB system 400 to crawl the SaaS provider 402. The file crawlingprocess 500 includes, for a first run (step 501), fetching admin logsfor file-related activities for a tenant in batches (step 502),processing the batch for unique file entries in the batch (step 503),pushing the file info into a queue (Q) (step 504), repeating steps 503,504 until the entire log is crawled (step 505), and storing the log'sstream-position for a next Run (step 506).

For a run X (step 501) where X is an integer greater than 1, the filecrawling process 500 includes, fetching admin logs from the laststream-position for a tenant (step 507), processing the batch for uniquefile entries in the batch (step 508), pushing file info to a queue (Q)(step 509), repeating through above steps 508, 509 for all tenants untilthe entire log is crawled (step 510), and storing the log'sstream-position for a next Run (step 511).

Crawling Based on the Breadth-First Traversal

FIG. 7 is a flowchart of a file crawling process 550 based onbreadth-first traversal. The file crawling process 550 contemplatesimplementation by the CASB system 400 to crawl the SaaS provider 402.For example, some SaaS providers 402 may not maintain a change log for atenant, but instead, provide a snapshot of a user's filesystem and thena change log for every user. The file crawling process 500 includes, fora first run (step 551), crawling through the entire list of entities(Users/Groups/SharePoint Sites) for a tenant (step 552), and storing thelist of entities for each tenant against and update Run # (step 553).For each entity, the file crawling process 550 includes, crawlingthrough the File System and capturing the list of files (step 554),storing the last delta link for every entity (step 555), and pushing thefiles in a queue (Q) (step 556). The file crawling process 550 includes,after the last user, the last file pushed in the queue (Q), updatingtenant info about Run # completion (step 557), and repeating through theabove steps for all tenants (step 558).

For run X (step 551) where X is an integer greater than 1, the filecrawling process 550 includes fetching a list of entities for the tenantfrom a store (step 559), for each entity, crawling through the FileSystem and capture the list of files (step 560), storing the last deltalink for every entity (step 561), pushing the files in a queue (Q) (step562), after the last user last file pushed in the queue (Q), updatingtenant info about Run # completion (step 563), and repeating throughabove steps for all tenants (step 564).

Flow Diagram

FIG. 8 is a flow diagram of example operations between the CASB client460, the controller 450, the message broker 422, a worker 430, and theSaaS provider 402. A new configuration or reconfiguration is provided,via the CASB client 460, the cloud-based system 100, etc. (step 601),and organization (tenant) information and credentials are provided tothe controller 450 (step 602). The controller 450 gets events and pushesthem in a queue (Q). The message broker 422 is configured to dequeue (DQ) the events and assign it to the worker 430 (step 604). The worker 430is configured to interact with the SaaS provider 402 to get admin events(step 605), which are provided as JavaScript Object Notation (JSON)events (step 606.1). The process is continued until the queue isemptied, the last event in the queue (step 606.2, 606.3).

The worker 430 can add new events in the queue, and the broker 422 candequeue the new events when assigning back to a worker 430 (step 607).The worker 430 gets file info (step 608) and receives JSON file infofrom the SaaS provider 402. The worker 430 can scan each file in thequeue (step 609), provide results to the controller 450, which dequeuesthe scanned file (step 610).

The controller 450 can provide results of the scan to the CASB client460, which returns information (step 611). The controller 450 can createa scan file (step 611.1) and receive a post-action (step 611.2) from theCASB client 460. For example, the CASB client 460 may perform DLP, andthe action can be allow, delete, quarantine, etc. The controller 450 canimplement the policy action in the queue (step 612), the brokers 422 candequeue the policy action (step 613) and assign the action to the worker430 which posts the action in the SaaS provider (step 614). The worker430 can provide the action result in a queue (step 615), the broker 422can dequeue the action results (step 616) and post the action result inthe log (step 617).

Webhook Integration

A webhook in web development is a method of augmenting or altering thebehavior of an application with custom callbacks. These callbacks may bemaintained, modified, and managed by third-party users and developerswho may not necessarily be affiliated with the originating website orapplication. Webhooks are user-defined Hypertext Transfer Protocol(HTTP) callbacks that can be triggered by some event, such as in a SaaSapplication detecting modification of content. When that event occurs,the SaaS application makes an HTTP request to the URL configured for thewebhook. Users can configure them to cause events on one site to invokebehavior on another. SaaS applications from the SaaS providers 402 areconfigured to support webhooks. Webhooks operation can be compared toAPIs. In APIs, one pulls data from a provider. In Webhooks, the SaaSprovider 402 pushes data out, e.g., the present disclosure utilizeswebhooks for identifying real-time modifications of content in SaaSapplications.

The present disclosure is described with reference to the CASB system400, including all of the various architecture implementations describedherein. Again, the CASB system 400 can be provided using the cloud-basedsystem 100, in another cloud (e.g., private cloud, public cloud, hybridcloud, etc.), as one or more servers 200 (e.g., an appliance locatedon-premises with an enterprise, off-premises, etc. as well as collocatedwith the SaaS providers 402). Note, the CASB system 400 as describedherein is configured to provide CASB functionality, which may also bereferred to as a CASB service. Again, the CASB service functions betweenthe SaaS service users and the SaaS applications from the SaaS providers402. Among the many services, CASB provides the most critical onesinclude identify and protect malware and policy enforcement (DLP).

The present disclosure contemplates an organization (i.e., a tenant, acorporation, an enterprise, etc.) having multiple SaaS applications andmultiple users spread across multiple countries, states, cities, etc.(generally geography) providing protection as soon as possible with theCASB system 400. That is, the CASB system 400 is configured to scandata, prevent data loss, etc. at near real-time for critical protection.To that end, the CASB system 400 is configured to detect changes in dataas soon as a user 102 is active and modifies the data. Further, the CASBsystem 400 is configured to scan data closest to the source, usinggeolocation, and scanning in compliance with local law and regulations.

CASB System with Webhooks Integration

FIG. 9 is a flow diagram of an architecture of a CASB-webhooks system700, and FIG. 10 is a flowchart of a CASB-webhooks integration process702, which may be implemented through the CASB-webhooks system 700, orin other approaches. The CASB-webhooks system 700 and the CASB-webhooksintegration process 702 operate with the constraint that there is noreplication of user data. The CASB-webhooks system 700 includes variouscomponents including a webhook controller 704, a gateway controller 706,an initiate subscription component 708, a data stream processing system710 such as Apache Kafka, a webhook listener component 712, asubscription helper component 714, a service publish component 716, anda process notification component 718.

Note, these various components 704-718 can be physical or logicalcomponents that are part of the CASB-webhooks system 700 and perform theCASB-webhooks integration process 702. The components 704-718 can beexecuted on one or more servers, including physical servers, virtualmachines (VM) executed on physical hardware, etc. The components 704-718can be integrated into one another, and FIG. 9 is illustrated to showfunctional operation. Those skilled in the art will recognize variousapproaches are contemplated. Also, in an embodiment, the components704-718 can be separated from the CASB system 400. In anotherembodiment, some or all of the components 704-718 can be included in theCASB system 400. In a further embodiment, some or all of the components704-718 can be included with the SaaS providers 402. Of course, acombination of any of these approaches is also contemplated. Also, anyof the components 704-718 may include one of the worker 430 in the CASBsystem 400, as well as a worker 430 outside of the CASB system 400.

The CASB-webhooks integration process 702 includes registration (step702-1), renewal (step 702-2), notifications (step 702-3), and triggering(step 702-4). Variously, the notifications step 702-3 and the triggeringstep 702-4 can be implemented by the components 704, 706, 716, 710, 712,718, 710 in communication with the SaaS provider 402 and the CASB system400 (bottom half of FIG. 9 ), and the components 708, 706, 710, 712, 714in communication with the CASB system 400 (top half of FIG. 9 ) can beused to manage the subscriptions, i.e., the registration step 702-1 andthe renewal step 702-2.

Generally, the registration step 702-1 involves identifying the users102 of a tenant, and this can be a process where IT admin performsconfiguration through the CASB system 400. The registration step 702-1can further include specifying monitoring functions per user, per group,per tenant, etc.

FIG. 11 is a flow diagram of subscription and renewal process 750 forthe registration step 702-1 and the renewal step 702-2. The subscriptionand renewal process 750 includes a tenant (or a provider) providing aconfiguration (step 750-1). This can be through the central authority152, the CASB client 460, etc. The configuration can include a tenantidentifier, user identifiers, actions for monitoring, content formonitoring, etc. The configuration is provided to a subscriptioncontroller (step 750-2). The subscription controller can publish theconfiguration to a subscription queue (step 750-3), which connects to aconsumer subscription listener (step 750-4) that connects to asubscription service client (step 750-5). The subscription serviceclient interfaces the SaaS provider 402 via batch API calls and receivesresponses for managing the subscriptions and configuration. This datacan be stored in a database 754.

Generally, the renewal step 702-2 involves periodically renewing thesubscriptions based on the timing of the SaaS provider 402. The renewalstep 702-2 operates based on a timer for each different SaaS provider402 from a scheduler (step 750-6). The scheduler publishes to thesubscription queue (step 750-7), which connects to the consumersubscription listener (step 750-8). The consumer subscription listener,which can be one of the workers 430, is configured to get subscriptionsthat need to be renewed and to make batch API patch requests for renewalto the SaaS provider 402 (step 750-9). For subscription and renewal, theinteraction between the SaaS provider 402 can be via controller 706.

Referring back to FIG. 9 , generally, the components 708, 706, 710, 712,714 communicate with the CASB system 400 for managing the subscriptions.The notifications step 702-3 includes receiving multiple notificationsvia webhooks that certain users and modifying certain content. Thenotifications can identify the user, the content, the event type (save,delete, add, etc.). The webhook notification is provided from the SaaSprovider 402 based on the subscriptions to the webhook controller 704,which connects to the gateway controller 706, which publishes thenotifications (service publisher 716) to the data stream processingsystem 710. The webhook listener 712 detects notifications from the datastream processing system 710 and causes a process notification 718,which can also be provided to the data stream processing system 710, foraction by the CASB system 400.

The triggering step 702-4 generally includes taking the notificationsfrom the data stream processing system 710 that were caused by webhooksand acting upon them in the CASB system 400. This action can include anyof the monitoring and scanning functions described herein. Of note, thenotifications step 702-3, and the triggering step 702-4 can be used bythe CASB system 400 for detection of which users 102 to process, toidentify the event type and process with delay or process instantly,etc. Further, the notifications step 702-3, and the triggering step702-4 can be used by the CASB system 400 to identify which queue to pushinto.

Geolocation

FIG. 12 is a map illustrating an example system including two CASBsystems 400A, 400B geographically distributed, and two SaaS applications800A, 800B also geographically distributed. In this example, the CASBsystem 400A, 400B are single nodes in a composite CASB system 400.Scanning data via one of the CASB systems 400A, 400B includesdownloading the data and performing policy actions or malware detection.The SaaS application 800 users in an organization can be locatedanywhere, and they download and upload data while using the SaaSapplications 800.

Similarly, the SaaS applications 800 can distribute and store dataacross the globe. Thus, strategically downloading data is critical forfast actions and remediation. To achieve this, the present disclosureincludes the CASB system 400 scanning data closest to the source, whichis most of the time near the location of the user. For implementation,the CASB system 400 detects the geolocation of the users 102 via theiruser devices 300 and routes the scan requests to the closest CASBscanners, namely the CASB systems 400A, 400B. Geolocation of the users102 can be fetched periodically.

Historical and Live Scanning Process

FIG. 13 is a flowchart of a historical and live scanning process 850 forCASB functionality. Tenants (customers) require scanning of historicaldata as well as live data that is being modified by users. Historicaldata is scanned, with the CASB system 400, by crawling the SaaSapplication using APIs. As described herein, the entities to be scannedare pushed into queues to be scanned. Along with historical data, thepresent disclosure can process live data modifications using webhooks.Notifications on changes are received, as described herein, and pushedinto the queues. The CASB system 400 can then prioritizes the livemodified data and perform the CASB functions. Specifically, historicalscans are important, but they can tolerate latency. Live datamodification requires scanning at near real-time speeds to detectproblems and to limit the impact on the user 102 (i.e., the user 102does not want to wait for each file, etc.).

The historical and live scanning process 850 can be implemented as amethod, in a server 200, and as computer-readable code stored in anon-transitory computer-readable storage medium. The historical and livescanning process 850 includes causing a scan by the CASB system of aplurality of users associated with a tenant in a Software-as-a-Service(SaaS) application where the scan includes any of identifying malware incontent in the SaaS application and identifying confidential data in thecontent in the SaaS application (step 851); during the scan which iscovering historical data in the SaaS application, receivingnotifications of the content being actively modified by any of theplurality of users (step 852); and including the content being activelymodified in the scan with the historical data (step 853).

The historical and live scanning process 850 can further includemaintaining geolocation of the any of the plurality of users (step 854);and causing the content being actively modified in the scan to beprocessed by the CASB system based on the geolocation (step 855). Thehistorical and live scanning process 850 can further includeprioritizing the content being actively modified in the scan higher thanthe scan of the historical data. The historical data can be scanned viaApplication Programming Interfaces (APIs) associated with the SaaSapplication, and the notifications of the content being activelymodified are via webhooks from the SaaS application.

The historical and live scanning process 850 can further include causingan action in the SaaS application based on the scan and based on policyand the content. The action can include any of allowing a file, deletinga file, quarantining a file, and providing a notification. Thehistorical and live scanning process 850 can further include causing theexecution of a file of the content in a sandbox for the identifyingmalware. The historical and live scanning process 850 can furtherinclude causing queueing of the content being actively modified and thehistorical data.

CASB In-Memory Data Store

As described herein, the CASB system 400, such as implemented via thecloud-based system 100, is configured to perform scans of data locatedin the SaaS providers 402. Again, the data can include files, email,etc., and the data can be accessed by the CASB system 400 via APIs. Thescan is for security and/or DLP policy, configured by a tenant. Asdescribed herein, if there is a DLP violation and/or if a file/emailcontains malware, this is referred to as a CASB incident or simply anincident. The present disclosure provides logging and reporting forincidents (as opposed to entire scan results). For CASB incidents, afile in the SaaS provider 402 can be modified and rescanned again andagain multiple times. The logging and reporting requirement forincidents are different from the regular weblog and firewall log wherethe previous transactions are immutable, while incidents change. Theobjective here is to provide tenant IT with the current snapshot of theincidents for analysis thereof. The CASB in-memory data store describesa highly scalable and efficient incident reporting approach for thelatest snapshot for a scan (for each file and email). There is a singlelog line recorded for each file/email that was updated when thatparticular file/email was last scanned.

FIG. 14 is a block diagram of a CASB in-memory data store system 900.The system 900 includes computing clusters 902 with various data nodes904. The data nodes 904 are where the actual data store resides. Acompany (tenant) can reside in one or more clusters 902 at the sametime. Also, a company can historically reside in one cluster 902 andmigrate to another cluster 902 as part of load balancing for theclusters 902.

The CASB in-memory data store system 900 includes data forwarder/routernodes 906 and query forwarder/merger nodes 908. The dataforwarder/router nodes 906 are configured to route logs, obtained duringa scan with the CASB system 400 and based on an incident, to appropriatecluster 902. The data forwarder/router nodes 906 perform the routingfunction for incidents. Note, the data forwarder/router nodes 906 haveintelligence, i.e., processing capability to receive a log and todetermine which cluster 902 to forward it to. Note, the CASB system 400can be multi-tenant, and the data forwarder/router nodes 906 know thetopology for each tenant, and this is used to forward a particular log,such as based on metadata identifying a tenant, to the cluster orclusters for that particular tenant. Also, if a company (tenant) isspread across multiple clusters 902, the data forwarder/router nodes 906can be configured to provide a log to all of the multiple clusters 902,such as via metadata. For example, in FIG. 14 , assume there are twoclusters 902 A and B, if a tenant is on both, the data forwarder/routernodes 906 attempt to distribute the log (record) equally among eachcluster 902. In an embodiment, the receives the log for each incidentfrom a worker 430 in the CASB system 400.

The query forwarder/merger nodes 908 are configured to receive queriesfrom an external source and performs such queries through the data nodes904. All requests (queries) land on query forwarder/merger nodes 908 anda node, based on a metadata of the tenant to cluster mapping, performsquery planning, sends requests to clusters 902, and, if the query hasbeen sent to multiple clusters 902, it will merge the results and sendresponses back to the client (requestor). Similar to the dataforwarder/router nodes 906, the query forwarder/merger nodes 908 havethe state to route the queries to a particular cluster 902, includingmultiple clusters 902 where a tenant is spread. Once the queryforwarder/merger node 908 receives a response, it merges the results ifthere where multiple clusters 902, and sends the result back to theclient.

Data Store Design and Data Structure

The data store in the CASB in-memory data store system 900 supportsvarious operations including an insert operation, a delete operation, anupdate operation, and a recovery mechanism to rebuild the data store incase of a service restart. FIG. 15 is a flowchart of a record processingprocess 950 implemented in the data store. The process 950 describes howupdate, delete, and insert operations are performed.

A new record is received (step 951). In case of any new record (step951), the process 950 includes checking if entry exists in a filehash ina data store 960 (step 952). If so (step 952), it means the process 950has seen that file earlier, and the old record is deleted from thefilehash and deleted from the data store in a deferred delete, describedherein (step 953). If the file does not exist in the filehash in thedata store 960 or after the delete step 953, the record is inserted inthe filehash in the data store 960 and in the data store (step 954). Fora recovery mechanism, the process 950 can include periodically flushingthe data store logs to a file (step 955). This can be every so often,based on a trigger event, etc.

The data store is mutable in-memory for highly-optimized updates andreports, for the most common dimensions of filtering, aggregation, andpagination. In the CASB system 400, the most common dimensions areapplication name (appname or application ID) and tenant name (i.e., theuser 203). As such, all reports are around these dimensions.

FIG. 16 is a diagram of an example implementation of the filehash in thedata store 960. The filehash in the data store 960 is stored in the datastore and can be a multilevel hash that includes a hash of a customerID, application ID, tenantname ID, etc., where data is hashed bycompany, application name (appname or appid), and tenant name(tenantname or tid). At the leaf level, a list of actual records isstored. This data store will always contain the latest incidentinformation for any file and for a particular combination of (appname,tenant, filename), only one entry is present, referred to as a datanodein FIG. 16 .

Also, the following facts are noted and assumptions made. A file oremail that is being scanned can belong to only one tenant (user), and atenant can belong to one application. The maximum scan durationsupported by the data store is for a certain period of time. Older scanresults will be cleaned up periodically by the product. Most reportscome with company, appname and tenant filters. Majority of theaggregation queries are on the filenames/email. So, the data storeoptimizes these reports.

The following provides pseudo code for the filehash in the data store960:

1) New rec received 2) Check if already same file already exist forcid,appid, tid combination In filehash  filehash =smhashapi_keymd5_old(filenamep, filelength);  head=&smslave_file_hash_table[(datanode− >filehash)%SMSLAVE_FILEHASH_SIZE]; _TAILQ_FOREACH(entry, head, filehash_entry) { If (cid, appid and tidmatches) /* delete old record if it is not referenced both from thistable as well as from inmemory /* data store  } 3) #definePREPARE_KEYREC(_cid, _appcat, _appid, _tid)  keys[keycnt++]._u_int=_cid;  keys[keycnt++]._enum = _appid;  keys[keycnt++]._u_int = _tid;\This will be used to find or insert record at which leaf node that is tobe put in the in- memory data store 4) Find leaf node forzks_prefix_tree_schema_get(ctx−>ptp_dstore, keys) for keys prepare instep (3) This will give us the head of list where actual complete recordis stored. 5) _TAILQ_INSERT_HEAD(head, datanode, comp_list_entry Thiswill make sure records are stored in descending order of time.

Data Duplication Prevention

In the CASB in-memory data store system 900, data is distributed acrossthe nodes 904 in clusters 902 based on some hash functions (e.g., key %noOfnodes) and records based on some key that can also beupdated/deleted in mutable systems. But when any node 904 is added ordeleted, the hash function can give different nodes for a key which wasearlier sent to node A and later the hash function sends to another nodeB leading to multiple copies of the same mutable record being ondifferent nodes 904, leading to inconsistency in data. This scenario canoccur when a company moves from one cluster to another cluster as partof the load balancing.

FIG. 17 is a diagram of data duplication prevention in the data store.To make sure that the only one record of a key lies across all the nodes904 in a cluster 902, whenever any node addition happens, a broadcast issent to delete that key to all of the nodes 904 in the cluster 902except the current node which the hash function distributes to. In thatway, all nodes 904 which have an old record of same key will deletetheir record and only one node will have the record.

Suppose in the CASB in-memory data store system 900, there is data witha key as FILEID distributed uniquely among 3 nodes (Node 1, 2, 3).FileId 4 belongs to Node 2. Now, Node 4 is added and due to this, thedata forwarder forward it do Node 4 leading to multiple copies of FILEID4. During this time, the data forwarder/router node 906 detects thatnode addition has been done in last X days and there can be chances thatthis data might be lying on other nodes, the data forwarder/router node906 sends a DELETE record of that key (in this case FILEID 4) to allother nodes which might be having this older record and each nodemanager handles the delete record in Deferred delete way which iscovered next.

Deferred Delete on Data Manager

Data managers cannot delete a record blindly without checking areference for that record otherwise it will show inconsistent recordwhen query/report runs. A deferred delete includes, suppose a query isrunning and referencing the data store, and a delete record is receivedwhere a query is also running. The data manager checks if the record isreferred by a query manager, and if yes, puts that record into adeferred list and marks it a soft delete. Whenever the reference of thatrecord becomes zero, the data manager deletes that record.

For insertion, it is checked from another file-based hash whether anentry is already there in the data store, and, if yes, delete thatrecord using the deferred delete operation and insert the new one.Insertion is fast as the process just needs to find a particular tenanthash bucket and insert at the head.

Benefits of Maintaining a Unique Key Across Nodes

In distributed query engines, most of the expensive queries are relatedto count distinct queries on high cardinality fields with aggregates.Consider the use case for the following query:

“from CASB select count(distinct filename) where companyid=1 group byappname, tenantid”

The above query is an expensive query as it needs to maintain the listof all files in memory to resolve duplicates and count the filenamesafter all data has been read. Due to the nature of the data store(multi-level hash), this complex count distinct query on unique keys canbe converted into a normal count query leading to faster response time.Thus, the query is converted to

“from casb select count(1) as unique files where cid=1 group by appname,tid”

Instead of maintaining the entries for each file, it just needs to readand count the rows as duplicates are removed at the time of recordupdate.

Reporting Design and Optimizations

A query request can come to query forwarder/merger node 908 forfollowing kind of companies:

a) Company which is not distributed across multiple clusters 902. Inthis case, requests need to be processed from an in-memory data storelying on a single data node and the node 908 acts as a forwarder forthese kinds of queries.

b) Company which is distributed among multiple clusters 902. In thiscase, requests need to be processed from many in-memory data store lyingon different clusters 902 and the node 908 acts as query planner andmerger for these kinds of queries.

For all these requests, optimized filtering and aggregation operationsare needed to have faster response time and reduce the resourceutilization.

Filtering

The most common reports require filters on appname and tenant name. Thedata store is maintained such that if any such filter comes, it ispossible to apply filtering at the hash level instead of reading allrecords. This will be applied at the hash level and once level filteringis passed, data can be directly passed to the consumer without applyingthe same filter again and again on each record.

Advantageously, a chain of records is filtered by applying a filter atlevel of in memory data store 960 skipping filter check for each record.This technique of using in-memory data store 960 for filtering can bedone for other use cases.

Inbuilt Cache for Popular Reports (Group by Appname, Tenantname)

The data is stored in a multi-level hash (FIG. 16 ) that includescompany, application and tenant nodes. These nodes are the parent tounderlying child nodes that contain the incident scan results. Thesenodes can work as a cache to store the summary of the underlying logs.For example, each application node can store the following information:

i) Count of malware incidents in the underlying nodes, e.g., from CASBselect count (transactions) where malware >0 where appname=BOX group bytenant;

ii) Count of DLP violation, e.g., from CASB select count (transactions)where DLP>0 group by appname, and

iii) number of files in the tenant.

The count on the parent nodes are updated when a new incident isadded/modified in the data store 960. The data update operation will benegligible because each file belongs to only one <company, appname,tenant> combination.

Since the data store 960 supports the scan reports for the last X days(e.g., 90 days), the parent nodes can store the counts for each separateday by maintaining X buckets. All the reports starting with day boundarytime can be served from the parent node cache.

Pagination for Aggregate Reports

In a distributed data system, running aggregation based queries on highcardinality fields can be very expensive. For example, from CASB selectcount (distinct filename) as trans group by ownername limit 25 offset 0.Here, the filename can have high cardinality in the range of millions,the ownername can have medium cardinality in range of 100's of thousand,and, in this query, a user wants to see 25 aggregated records.

The query forwarder/merger node 908 can send this query to all theindividual data nodes 904, which can each have a query manager, and thequery manager processes the query as follows:

1) start reading data and create hash based on ownername column;

2) for each hash bucket (unique ownername values), maintain uniquedistinct filename values; and

3) once data read is finished, take 25 bucket and count unique A valuesand send results.

In this approach, even if the query forwarder/merger node 908 asks for25 aggregate records, the query processor is processing all 100k uniqueB entries and maintaining a hash for all the buckets leading to queryslowness. Also, the hash lookup for all the values of aggregating columnwhich will be processor intensive and maintaining hash for all thevalues of aggregating column will take too much memory.

To solve this issue, the query forwarder/merger node 908 forwards therequest to each query processor on each node 904 and asks for 25 recordsfrom each query processor. The query processor maintains a hash for 25entries instead of maintaining a hash for all the 100K unique entries

The query processor maintains metadata of the aggregated column which ithas seen previously. The metadata is used to skip previous run responseentries which the query processor has already sent. When this request isreceived the first time with offset 0, the metadata will be empty. Thefollowing steps for the first run are as follows:

a) start reading data and create hash based on ownername column.

b) once a hash contains 25 unique entries (limit count), the querymanager creates a filter with these values and starts dropping entriesfor the rest of (100k−25) entries instead of maintaining hash and doinglookup. This will help in reducing memory footprint and query faster.

c) once the query is finished on the query node, each query node willsend its result to the query forwarder node 908 and each query processoron that node maintains metadata of these unique entries for the nextpagination request if it comes.

The steps for the second run where a client requests the next 25records:

a) the query processor starts reading data and puts the entry into thehashtable only if it does not belong to metadata entries (previous runresponse unique entries);

b) now when a hashtable contains unique 25 entries, the query managercreates a filter with these values and starts dropping entries for therest of entries instead of maintaining hash and doing lookup;

c) once the query is finished on the query node, each query node willsend its result to the query forwarder and each query processor on thatnode updates metadata of these unique entries for the next paginationrequest if it comes; and

d) now metadata will have 50 entries (25 for 1st run and 25 for 2ndrun).

The same can continue for the next pagination request.

Advantageously, this reduces the memory footprint for the query engineas there is no need of doing aggregation for all the values of adimension, and the response time will be faster.

For queries spanning multiple in-memory data store nodes 904, the queryforwarder/merger node 908 will merge the results and return the resultto the client. For a single in-memory data store, the queryforwarder/merger node 908 will forward the result to the client.

Stateless Query Execution Engine with Cached Support

For all expensive queries, the results can be cached after the queryexecution. The cache could be used by the subsequent queries in thefollowing cases:

i) a similar query is received from another user and the cached resultscan be served directly to the client.

ii) The cached data structure can be further processed to produce thereport. In this case, the query will be faster as it will be working ona subset of data rather than a complete data store.

For example, suppose the first query is

“from CASB select count(distinct filenames) group by appname, tid”

In this, once the query is finished, the results are sent and saved incache with an Abstract Syntax Tree (AST) acting as metadata for querymatching.

For example, suppose a second query is

“from CASB select count(distinct filenames) where appname=BOX group bytid”

Once the query is parsed, the query engine will try to check based onincoming Query AST (Abstract Syntax Tree) by matching it previous queryAST's and see if it matches. In this example, group by clause AST match,cache contains appname and tid and the query contains tid only so thisquery can be run as subset of original query.

If the select clause also matches, where the clause of the 2nd query isalso a subset of original query as original query contains data foreverything and also appname field in where clause is present in group byAST of cache query (appname). In this case, the query will be fasterbecause it is not reading all the data, it is just reading the aggregateresults and from there deriving response for this query result.

For example, suppose a third query is

“from CASB select count(distinct filenames) where ownername like(\“abhishek\”) group by tid”

Here, the group by AST is subset of group by AST of query cache (query1), the select clause AST is subset of select AST of query cache (query1), and the where clause AST does not match cache AST because ownernamefield is not there in group by AST, select by AST of cache AST.

Algorithm

a) if a filter matches multilevel keys (appname or tid), it converts itinto level filter;

b) if it is a count distinct query on a unique key with a group bymultilevel keys convert it into a normal count query.

For example “from CASB select count (distinct filename) group byappname, tid,” this can be converted to from CASB select count(filename)group by appname, tid.

c) count based queries multi-level hash keys can be served from nodecount metrics instead of walking through multilevel hash leaf entries.In this, the query engine producer will be just passing the level keyand metrics to consumer instead of passing each record.

Sample Queries

from casb_fileapps select count(distinct filename) as tot_dlpincidentswhere appname=BOX and ANY(dlpdictids)!=0″→output: tot_dlpincidents=34.

from casb_fileapps select count(distinct filename) asunique_fileincidents, id2name(ownerid) as ownername group by useridownerid limit 100″→output ownerid,unique_fileincidents,ownername,65761,130,traffic_loc->other, 65786,1,ca_loc.

from casb_fileapps select time as time, filename as filename, appname asappname id2name(tid) as tenantname, dlpdictids where ANY(dlpdictids)!=0limit 2″→output 1594175160, one.txt, BOX, ca_loc, [61,62], 1594167110,casb_test.txt, GDRIVE, g_tenant, [1,3].

Although the present disclosure has been illustrated and describedherein with reference to preferred embodiments and specific examplesthereof, it will be readily apparent to those of ordinary skill in theart that other embodiments and examples may perform similar functionsand/or achieve like results. All such equivalent embodiments andexamples are within the spirit and scope of the present disclosure, arecontemplated thereby, and are intended to be covered by the followingclaims.

What is claimed is:
 1. A method performed by a Cloud Access SecurityBroker (CASB) service, the method comprising steps of: scanning datastored in one of a cloud provider and a Software-as-a-Service (SaaS)application, wherein the data is for a user associated with a company ofa plurality of companies; detecting an incident in a file or email inthe data during the scanning; maintaining details of the incident in anin-memory data store, including a current snapshot of the file or email;and providing a notification to the tenant of the incident.
 2. Themethod of claim 1, wherein the steps further include: subsequent to theincident and while the file or email is being updated, updating thedetails of the incident in the in-memory data store.
 3. The method ofclaim 1, wherein the incident is one of a Data Loss or LeakagePrevention (DLP) violation and malware.
 4. The method of claim 1,wherein the steps further include: periodically storing records in thein-memory data store in a file for recovery.
 5. The method of claim 1,wherein the in-memory data store includes a multi-level hash for arecord, for the incident.
 6. The method of claim 5, wherein themaintaining includes performing any of inserting, deleting, and updatingthe record, for the incident.
 7. The method of claim 5, wherein themulti-level hash include at least a hash based on the company and a hashbased on an application name.
 8. A non-transitory computer-readablemedium comprising instructions that, when executed, cause one or moreprocessors to implement a Cloud Access Security Broker (CASB) service,the CASB service comprising steps of: scanning data stored in one of acloud provider and a Software-as-a-Service (SaaS) application, whereinthe data is for a user associated with a company of a plurality ofcompanies; detecting an incident in a file or email in the data duringthe scanning; maintaining details of the incident in an in-memory datastore, including a current snapshot of the file or email; and providinga notification to the tenant of the incident.
 9. The non-transitorycomputer-readable medium of claim 8, wherein the steps further include:subsequent to the incident and while the file or email is being updated,updating the details of the incident in the in-memory data store. 10.The non-transitory computer-readable medium of claim 8, wherein theincident is one of a Data Loss or Leakage Prevention (DLP) violation andmalware.
 11. The non-transitory computer-readable medium of claim 8,wherein the steps further include: periodically storing records in thein-memory data store in a file for recovery.
 12. The non-transitorycomputer-readable medium of claim 8, wherein the in-memory data storeincludes a multi-level hash for a record, for the incident.
 13. Thenon-transitory computer-readable medium of claim 12, wherein themaintaining includes performing any of inserting, deleting, and updatingthe record, for the incident.
 14. The non-transitory computer-readablemedium of claim 12, wherein the multi-level hash include at least a hashbased on the company and a hash based on an application name.
 15. Acloud-based system comprising: a plurality of nodes communicativelycoupled to one another, wherein the plurality of nodes include one ormore processors configured to implement a Cloud Access Security Broker(CASB) service, where one or more nodes are configured to scan datastored in one of a cloud provider and a Software-as-a-Service (SaaS)application, wherein the data is for a user associated with a company ofa plurality of companies; detect an incident in a file or email in thedata during the scanning; maintain details of the incident in anin-memory data store, including a current snapshot of the file or email;and provide a notification to the tenant of the incident.
 16. Thecloud-based system of claim 15, wherein one or more nodes are furtherconfigured to: subsequent to the incident and while the file or email isbeing updated, update the details of the incident in the in-memory datastore.
 17. The cloud-based system of claim 15, wherein the incident isone of a Data Loss or Leakage Prevention (DLP) violation and malware.18. The cloud-based system of claim 15, wherein one or more nodes arefurther configured to: periodically storing records in the in-memorydata store in a file for recovery.
 19. The cloud-based system of claim15, wherein the in-memory data store includes a multi-level hash for arecord, for the incident.
 20. The cloud-based system of claim 19,wherein the maintaining includes performing any of inserting, deleting,and updating the record, for the incident.